DOCUMENT RESUME 



ED 349 788 



FL 019 993 



AUTHOR 
TITLE 



PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Peretz, Arna S.; Shoham, Miriam 

Does Topic Familiarity Affect Assessed Difficulty and 
Actual Performance on Reading Comprehension Tests in 
LSP? 
90 

I8p.; Paper presented at the Annual Meeting of the 
Academic Committee for Research on Language Testing 
(9th, Kiryat Anavim, Israel, April 25-27, 1990). 
Reports - Research/Technical (143) — 
Speeches/Conference Papers (150) 

MF01/PC01 Plus Postage, 

Advanced Students; College Students; '"'Difficulty 
Level; English (Second Language); Foreign Countries; 
Higher Education; '"'Languages for Special Purposes; 
Language Tests; ''Majors (Students); '"'Prior Learning; 
"Reading Comprehension; Reading Tests; Second 
Language Learning; Second Languages; Student 
Attitudes; ''Technical Writing 
Ben Gurion University (Israel) 



ABSTRACT 

A study investigated the hypothesis that topic 
familiarity and assessed difficulty of a second language text 
correlated positively with performance on reading comprehension tests 
in languages for special purposes (LSP). Subjects were 177 advanced 
students of English as a Foreign Language (EFL) at Ben Gurion 
University (Israel). Faculty from the schools of Humanities and 
Social Sciences (HSS) and Science and Technology (ST) were asked to 
assess the difficulty of an HSS-related text and a ST-related text 
from a college-level EFL reader* The texts were found to be of 
comparable difficulty and comprehensible to an educated layman. 
Students then took a reading comprehension test using the passages, 
and were asked to evaluate the passages' difficulty. Significant 
interaction between faculty and assessed difficulty of text confirms 
that EFL students find texts related to their fields of study more 
comprehensible than texts related to other topics. However, results 
of multiple-choice comprehension questions based on the texts 
indicate that students' subjective evaluation of the relative 
difficulty of a text is not necessarily a reliable predictor of their 
actual performance on reading comprehension tests. Based on these 
findings, it is argued that creation of many different reading tests 
on specialized topics at the university level is not justified. A 
24-item bibliography and tabulations of study results are appended. 
(Author/MSE) 
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DOES TOPIC FAMILIARITY AFFECT ASSESSED DIFFICULTY AND ACTUAL 
PERFORMANCE ON READING COMPREHENSION TESTS IN LSP? 



The hypothesis that topic familiarity and assessed difficulty or a text 
correlate positively with performance on reading comprehension tests in LSP 
was investigated in a study with 177 advanced students of English at Ben 
Gurion University. Subjects from the faculties of Humanities/Social Science 
(HSS) and Science/Technology (ST) were asked to assess the difficulty of a 
HSS-related text and a ST-related text. Significant interaction between 
faculty and assessed difficulty of text confirms that EFL students rate texts 
related to their field of study as being more comprehensible than texts 
related to other topics. The results of multiple-choice type comprehension 
questions based on the two texts indicate that students' subjective evaluation 
of the relative difficulty of a reading text is not always a reliable inoev ot 
their actual performance on reading comprehension tests The findings of this 
study have practical implications for testing reading comprenension in LSP 



With the growing tendency to structure EFL language courses to fit the 
specific needs and interests of students from different disciplines, the 
question of the nature of the text used in reading comprehension tests 



Arna S, Peretz and Miriam Shoham 
Ben Gurion University of the Negev 



ABSTRACT 



INTRODUCTION 
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assumes great importance. As Alderson and Urquhart (1983. 1 985a. 1965b) 
note, there are valid arguments both for and against the use of the content- 
related text as a vehicle for testing reading comprehension. It can be claimed 
that background knowledge in the content area of a reading passage can enable 
students to perform at their highest ability level on reading comprehension 
tests. On the other hand, tests which are too specialized may assess subject 
matter knowledge more than they test reading proficiency. Subject-related 
texts might also discriminate against individuals or groups who happen tu 
possess less background knowledge in a particular field. If different 
specialized texts are used to test students from various departments, the 
problem of comparable difficulty of texts and tests must also be considered. 

A question that has been considered in second and foreign language reading 
comprehension research is whether interest in or preference for a topic 
affects reading comprehension results. Although empirical evidence is 
limited, there is some Indication that interest in a topic does not necessarily 
correlate positively with grades on reading comprehension tests. 

Nelson (1 985) found no significant correlation between performance and 
preference on matched readings of American and Egyptian topics for Egyptian 
subjects. Williams' study (1983) showed that while high interest material 
enhanced comprehension, low interest material did not depress comprehension 
when compared to neutral interest material. These results are compatible 
with the findings of Stef fensen et al ( 1 979), Reynolds et al. ( 1 982). and 
Lipson (1984) who have suggested that partial knowledge can Interfere with 
comprehension if it conflicts with information in the text. Maria and 
MacGinitie (1981, 1982, 1983). Carrell and Eisterhold (1983). Alvermarm et 



al. ( 1 985), and Gordon ( 1 987) also concluded that incomplete or inaccurate 
prior knowledge or knowledge that is in conflict with the information 
presented in a text interferes with reading comprehension. 

Baldwin et al. (1985) investigated the effects of topic interest and prior 
knowledge on reading comprehension using seventh- and eighth-grade 
students. They found that although there was no correlation between topic 
interest and prior knowledge, it seems that prior knowledge and topic 
interest have an additive effect on reading comprehension. Using college 
students as subjects, Weber (1980) and Entin (1981 ) obtained results that 
indicated that interest and prior knowledge were correlated and that the two 
factors had an additive effect on comprehension. Baldwin et al. ( 1 985) 
suggest that these differences stem from adult perceptions and adult 
prerogatives. Whereas school children are forced to study topics in which 
they may or may not be interested but about which they may be 
knowledgeable, the situation is different for adults. As people get older, they 
tend to specialize; in such situations interest and knowledge may closely, 
correspond. 

The studies reviewed investigated the relationship between such factors as 
topical interest of the text, familiarity with the topic, prior knowledge 
related to the text topic, and performance on reading tests. They give no 
indication, however, as to whether tests based on content-related teAts 
would be assessed by subjects as being easier than tests based on other 
topics, although this Is implicitly assumed (Alderson and Urquhart 1985b), 
neither do they indicate whether such subjective assessment of test 
difficulty is a reliable index of actual performance on the test. The present 



study addresses this issue. The research questions were formulated as 
follows: 

1. Will students of EFL perform better on a reading comprehension test 
whose content is on a topic that is related to their general field of 
study than on a reading comprehension test whose content is on a topic 
that is related to another subject, given that the texts are of 
approximately comparable difficulty and comprehensible to the 
educated layman? 

2. Will students of EFL rate a reading comprehension test on a topic that is 
not related to their field of study as being more difficult than one 
whose topic is related to their field of study, given that the ..xts are of 
approximately comparable difficulty and comprehensible to the educated 
layman? 

METHOD 

Subjects 

The subjects consisted of 1 77 advanced students at Ben Gurion University of 
the Negev: 80 students in the Faculties of Science and Technology (ST) and 97 
students in the Faculty of Humanities and Social Sciences (HSS). 

Materials and Procedure 

Two passages were excerpted from articles that had appeared in the 
Humanities and Social Science (HSS) reader and in the Science and Technology 
(ST) reader. The articles in these readers, which are used in the EFL cour ses 
at Ben Gurion University, are taken from academic journals in fields related 
to the students' major subjects. The texts themselves, however, are not 
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highly specific, since the students are mostly first-year students and are 
thus only at the beginning stages of their academic studies, in addition, the 
EFL teachers themselves are not content-area experts. The passage excerpted 
from the HSS reader dealt with the ethical considerations of using human 
subjects in medical research; the passage from the ST reader dealt with wavs 
to reduce fuel consumption in automobiles. Both passages were 
approximately the same length and were judged by five experienced EFL 
teachers at Ben Gurion University to be comprehensible to the educated 
layman. 

The format of the tests was similar to tests administered during the school 
year. Both reading passages were presented to all the subjects. Each reading 
passage was followed by a set of fourteen multiple-choice comprehension 
questions which focused on general comprehension, recognition of referents, 
and ability to deduce vocabulary in context. In half the tests the HSS-related 
subtest preceded the ST-related subtest while in the other half the order was 
reversed. The two versions of the test were distributed randomly. After 
completing the test, students were asked to rate the difficulty of each 
subtest on a scale of 1-5, from very easy (1) to very difficult (5) (Appendix 
A). The use of a dictionary was not allowed. 

RESULTS 

The means, NTs and standard deviations for scores of each contei it-area teAt 
passage are presented in Table 1. 

Insert Table 1. about here. 
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In a two-way analysis of variance text by faculty the two main effects as 
well as the interaction were significant. Students from the Faculties of 
Science and Technology scored higher (X=74.6) than students from the Faculty 
of Humanities and Social Sciences (X=60) on the entire test battery as well as 
on each subtest [F, (1,165 = 69.75) P 0.000 1 ]. Mean score on the HSS-reiated 
subtest (X=70.41) was significantly higher than on the ST-related subtest 
(X=62.85) [F, (1,165 = 25.87) P 0.0001], The two-way interaction text by 
faculty was also highly significant [F,(1,165 = 19.08) P 0.0001], but while 
HSS students performed much better on the text from the HSS reader, there 
was no significant difference in performance on the two texts by ST students. 

The means and standard deviations of the assessed difficulty score for each 
passage by faculty are presented in Table 2. 

Insert Table 2. about here. 
A two-way analysis of variance text by faculty on the questionnaire yielded a 
significant main effect for faculty [F, (1,193 = 9.69) P 0.0025], but no 
significant main effect for text. In general, students from the HSS Faculty 
rated the entire test as being more difficult (X = 3.55) than students from the 
ST Faculties (X = 3.25). A two-way interaction text by faculty was alsu 
highly significant [F, (1,143 = 13.16) P 0.000 1 ]. HSS students rated the ST- 
related subtest as being more difficult than the HSS-related subtest (3.77 vs. 
3.33), while ST students rated the HSS-related subtest as being more 
difficult than the ST-related subtest (3.60 vs. 2.89). 

Test reliability computed for the entire test battery (Kuder Richardson 
formula 20 = 0.73) proved acceptable. 



DISCUSSION 



The answer to our first research question, will students perform better on a 
reading comprehension test whose reading content is on a topic that is 
related to their general field of study than on a reading comprehension test 
whose content is related to another subject, was inconclusive. As in a 
previous study (Shoham, Peretz, and Vorhaus 1987), students from the 
Faculties of ST obtained higher mean scores not only on the ST-related 
subtest but also on the HSS-related subtest, it is interesting to note that Koh 
(1985), In a study with several groups of students, also found that science 
students performed best on reading comprehension tests, even when the texts 
were related to fields of study other than their own. Koh (1985) attributes 
this success to the better language proficiency of the science students which 
could compensate for ignorance of the subject matter. In our case, although 
we did not test for language proficiency separately, the population In the 
Faculties of ST come from the same general secondary school system as the 
population in the Faculty of HSS. Indeed, a considerable percentage of the ST 
students come from vocational high schools in which, if anything, the level of 
English instruction Is lower than in the regular high schools. We therefore 
have no reason to believe that in our situation high language proficiency could 
account for the overall higher scores of these students. A possible 
explanation for the better performance of the ST students could be that the 
higher entrance requirements to these faculties translates into students with 
overall higher learning competence. 

Another possibility, although this is only a conjecture, is that ST students are 
trained to read more carefully In their content areas and are therefore more 
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discriminating readers. We have no way of ascertaining how and if students 
made use of whatever relevant background knowledge they possessed relating 
to the ST-related text. We can only note that they did not seem to be 
disadvantaged by being tested on a text outside their field of study. This can 
also be interpreted to mean that they did not seem to be especially 
advantaged when they were tested on a text related to their field of study. 
Our findings are compatible with the conclusions of both Clapham (1990) and 
Carrell (1983). Clapham (1990) found that "subject area had no significant 
effect on scores" (p. 11 ) and, in some cases, "students appeared, if anything, 
to do slightly better at the tests outside their field of study" (p. 1 1 ). Carrell 
(1983) noted that "nonnative readers show virtually no significant effects of 
background knowledge" (p. 183). It is suggested by Carrell (1983) that 
nonnative readers, unlike native readers, do not activate existing content 
schemata; in other words, they do not make the necessary connections 
between the text and appropriate background knowledge. 

In the case of the HSS students, their mean on the ST-relaled subtest was 
considerably lower than their mean on the HSS-related subtest (X-53 vs. 
X=66.6). it thus seems that the HSS students were disadvantaged by being 
tested on a text that was related to a totally unfamiliar field of study, even 
if the text did not require specialized knowledge for comprehension. The 
more general topic of the HSS-related text, which was not directly related to 
any specialized field of study, turned out to yield higher scores. 

The question whether students of EFL would rate a reading comprehension 
text on a topic that is not related to their field of study as being more 
difficult than one whose topic is related to their field of study was answered 



positively. There was a strong interaction text-topic bv faculty un difficulty 
rating. This seems to indicate that students feel more comfortable with a 
text on a topic that is related to their field of studies and that they feel more 
threatened by a text on a topic that appears unfamiliar. 

In the case of the HSS students, assessed difficulty and actual performance 
seem to match. But for students of ST, actual performance on the H5S- 
related subtest was about the same as on the ST-related subtest, although the 
former was rated as being considerably more difficult. These results can be 
interpreted in basically two ways. 

First, it is possible that students rated the relative difficulty, i.e. their 
degree of comprehension of the texts, accurately, but that the scores on the 
test did not present a true picture of their comprehension. This explanation 
is consistent with doubts expressed by researchers such as Shohamy (1984) 
and Alsanian (1985) about the degree to which reading comprehension tests, 
and especially the multiple-choice test format, measure comprehension 
rather than some other ability. 

Another interpretation of the results is that students did not assess text 
difficulty correctly. Carrell (1983) has already suggested that "nonnative 
readers appear not to have a good sense of how easy or difficult a text is for 
them to understand" (p. 183). In our case, they misjudged the degree of their 
comprehension either because they were mistakenly reassured in the case of 
the apparently familiar topic or unduly intimidated by the apparently (tun- 
familiar topic. Carrell ( 1 988) notes that ESL readers may "rely un prior 
knowledge to infer or guess what is likely to be in the text rather than 
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actually sampling or processing much of the text" (p. 108); this may also be 
true of EFL readers. This means that students implicitly assumed that they 
would know more about a topic that seemed familiar than they would about 
one which appeared less familiar. In fact, however, this was true only in the 
case of the HSS students, who really know very little about the different 
kinds of engines and the factors that affect fuel consumption. As to why the 
ST students rated the HSS-related subtest as being more difficult, although 
its content was quite general, we can only surmise that the word "ethics" in 
the title suggested to them that this passage was on a philosophical subject, 
i.e. something far removed from their field of study. 

CONCLUSIONS AND SUGGESTIONS FOR FURTHER RESEARCH 

The logical assumption that students prefer texts on topics that are, or 
appear to be, related to their field of study and that they rate tests based on 
such topics as being easier than tests based on topics that are related to 
other subjects is confirmed by the study. 

At the same time, the results indicate that students" "hunches" of how 
appropriate or fair a test is are not not always a reliable index of their 
performance on reading tests. We suggest that this may be either because a 
given test does not constitute a reliable enough measure of comprehension of 
the text or because the students' assessment of their own comprehension is 
affected by their own expectations that a test on a more familiar topic will 
be more comprehensible. The two interpretations are not necessarily 
mutually exclusive. 
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Our last finding, that scores obtained on tests based on a general or less 
familiar topic are comparable to scores obtained on tests based on a topic 
that was generally more familiar, has practical implications, it appears that 
the creation of many differentiated reading tests based on specialized topics 
in such "real" testing situations as final examinations in large academic 
institutions is not justified in terms of the time and effort entailed, 
especially if such tests are to be utilized on a "one-time" basis. 

Directions for future research that are suggested by this study are a finer 
analysis of test-task and question-types and the effect of more specialized 
texts on tests of reading comprehension. It would also be useful to use 
mentalistic measures in order to learn more about how subjects make use of 
textual and extra-textual informatics while answering reading comprehension 
tests. 
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Table 1. Means and standard deviations of scores* for each text by faculty 



Text Ethics of Experimentation 


Fuel Economy of 


Mean ^core 




(HSS) 


Light Vehicles (ST) 


for Faculty 


Faculty 








HSS 


X = 66.6 


X = 53.5 


X = 60 


n = 97 


Sd= 14.6 


sd= 17.1 




ST 


X = 74.9 


X = 74.3 


X = 746 


n = 80 


Sd= 13.5 


sd= 13.3 




Mean Score 


70.41 


62.85 




for Text 


sd 14 


sd 15.1 





* on a scale of 0-100 



Table 2. Means and standard deviations of the assessed difficulty score* 
for each passage by faculty 



Text Ethics of Expen- Fuel Economy of Mean Difficulty 
mentation (HSS) Light Vehicles (ST) Score of Entire Test 



Faculty 



HSS X = 3.3 
n = 80 sd= .76 


X = 3.8 
sd= .89 


X 


= 3.6 


ST X = 3.6 
n = 65 sd = .70 


X = 2.9 
sd= .75 


X 


= 3.2 


Mean Dlffl- X = 3.5 
culty Score 
for Text 


X = 3.4 






*onascaie of 1-5 
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Appendix A. Student questionnaire. 



Surname: 

First Name: 

I.D.: 

On a scale of 1 (very easy) to 5 (very difficult), rate the difficulty of "Fuel 
Economy of Light Vehicles". 

12 3 4 5 

very easy quite easy average quite difficult very difficult 



On a scale of 1 (very easy) to 5 (very difficult), rate the difficulty of "Ethics 
of Experimentation with Human Subjects". 

1 2 3 4 5 

very easy quite easy average quite difficult very difficult 



